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ABSTRACT * 

, ' The terms "rigor* and "relevance" most often surface 

in discussions of methodological adequady. Assessing epistemological 
■relevance is equivalent to answering the question, • "Is this 
particular research question worth asking at all?" Epistemological 
rigor refers to the properties of a "researchable" problem. If, one 
accepts +he proposition that different kinds of questions tequire 
different ki^ds of methodologies, then the question of methodological 
relevance can be asked as, "Is this pa.rticular method .(or aodell • 
appropriate to the questions that I am trying: to answer?" 
Methodological rigor is a determination of whether the method 
selected meets certain standards for a "good* or "trustworthy" 
method^ a "sound" design, or an*"appropj£a*te" type of data analysis. 
It is frequently argued tftat one must cffoose between rigorous and 
relevant methods, .This argument demonstrates a confounding o-f the 
notions of methodological rigor and methodological relevenze. Rigor 
and relevence are not ftecessarily inversely jrelated , These issues are 
beir.g discussed M^y vocational evaluators, and warrant additional * * 
investigation, (Btf) 
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DEFINING RIGOR AND RE LE VAN CE^IN^ 
VOCATIONAL EDUCATION EVALUATION 

The term3 'rigor 1 and 'relevance 1 most often surface in discussions of 
methodological adequacy, ■ If they have a. 'classical 1 meaning, then, m^st 
likely, the phrase 'rigor verfcus relevance* refers to the tradeoffs involved 
in designing an experiment that has both high internal validity (rigor) and 
hi<Jh external validity (relevance) (Campbell and Stanley, 1966), Within the 
context of the current methodological 'debate, these two terms have been 
used in a rather general way to characterize the differences between 'hard 1 
data and traditional, scientific, or' quantitative methodology which is* 
ri^prous and *soft* data an<3, less conventional , naturalistic, or qualitative 
methodology which is relevant, 

I initially intended to investigate vocational education evaluation * 
models, methods, and frameworks in view of their treatment of . these two 
dimensions of methodological adequacy. Yet, attempts to clarify the. meaning 
of the terms 'rigor/ and *relevance ; revealed that they have an epistemological 
as well as a methojjrflogical meaning. ^ 

Hence, in what follows, I propose a more expanded analysis of rigor and 
* relevance. -Four distinct, though interrelated, notions of rigor an& relevance 

are identified: epistemological relevance, epistemological rigor, method- 
m ological relevance, and methodological rigor. Each is explained and an attempt 
is made to illustrate the treatme^ of each in the 'literature on Vocational 
education evaluation. The paper concludes with a discussion of the need to 
furtter investigate each dimension. ^ 



ERLC 



Rigor and ReleVance 
2 



Defining Rigor and Relevance 

logical -and conceptual analysis of the potions of rigor and relevance 
in the context of spciai science research and evaluation reveals an ^ 
epistemological arid a methodological usage of eaph term.. These four aspects 
of rigor and relevance are explained below, 

Epistemological Relevance 

It is commonplace to characterize the major criteria for adequate research 
problems as relevance and fruitf ulness.. For example, steiner explains % 

that for an educational problem to be relevant it must be generative of 
scientific* philosophical, or praxiological knowledge about educatiori^ The 
mark of a fruitful problem is that it must be capable of leading to the 
extension of knowledge. Assessing epistemological relevance is thus equivalent 
>to answering the question, "Is this particular research question worth asking 
at all?' 1 \ ' 

% We nuyht extend this notioft of epistemolpgical relevance in research to 
the domain of evaluation in the following way. To determine whether a 
particular evaluation question is worth asking (or whether a particular 
evaluation is worth pursuing), we mu^P, first ajiswer twa questions; (1) What 
is to be evaluated?, ~an& (2) Why is it to be eValuated? These questions must 

be answered to the satisfaction of stakeholders in any given e^/aluati^n before 

* / 
questions of how to proceed are proposed. Failure tq adequately specify-the^ 

evaluand — jthe entity being evaluated (Scriveri, 1^79)--an*to identify the 

intended uses and users of evaluative information will'likely result ii> an 

inaccurate evaluation of little use to anyone/ H 



Determining epis topological relevance need not be y narrowly conceived _as 
a task unique to positi vrstic science: fhe following two approaches to 
assessing epistemo logical relevance originate in" quite different points of 
view regarding the nature of evaluation/" The first approach-, -proposed by" 
Joseph Who ley .and his colleagues at thef Urban Institute -(Wholly, 1975,^19^7; 
Horst, et al/, 1974) is compatible with the J traditional view of evaluation, 
as 1 evaluation- research* * The approagh, knowr^ as "evaluability assessment, M 
requires clarifying and defining the evaluai}3 from the perspectives' of both 4 
the *user and the evaluator. The major elements .of an evaluability assessment 
are characterized iiy Wholey (1977) afs:. ■ ■ * - ' ' 

+ ' I m * ' 

* ' * /- 1 ' ' t 

1* Determining -tt>e boundaries/ of the. problem/program, ,i,e. , , . , / * 
what is it that is tio be ahalyzed?, vihit "are. f the program 
objectives? * ' " " ^ 

2,_ Gathering inf ormation thafc def irw5S-program%bj£^tiyes, 1 
" activities, and underlying, assumptions. " " 

* i ■ ^ * + * h " * 

Develdpinij a model of program actitfiti&s and obje%?i^ei 

from the point of view cjf the intended 1 user *of the;V 
evaluation information., 

4* Determining to what ext&nt the defiivitipn fif the program," * * 
as containedin the model, m ip . sufficiently unambiguous to 
permit a useful evaluation. ■ . , ^ : 

1 5- Presentation of ^ the above information to 1 the" intended user ; 

of the ^Valuation and determination- of next steps to Ipe .taken. 

* * " ■ ■ * 

Though quite "antithetical to* the " general approach ^of evaluation research^' 

Guba's (1978) coirartentary^ on the methodology of naturalistic inquiry, in 

' ' i ♦ - " \ - , , v 

evaluation also addresses ther queaftipn 6f epAstemological relevance.- In 

* ■ ♦ - 

surfacing the concerns and'issues of "relevant parties to the evaluation, 

t the naturalistic investigator is efgaged fl iri a process of determining - what is 

to b& evaluated and why It is' to be evaluated. Having cycled through repeated- 
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I # 

' ■ ■ / 

phases of discovery and verification* the evaiuator possesses a preliminary 

set of categories of information,, Guiia suggests that considerations of salience 

credibility, uni<3ueness, heuristic vajue, feasibility, and materiality be 

employed to prioritize the categories and thereby focu^ the inquiry,' As a 

result'of this process,* the evaiuator is able to pursue those- categories o£ 

concerns and issues- "most worthy .of further exploration," 

It,should be apparent from the^se two examples that, regardless of one's 

philosophical orientation regarding the nature of evaluation , assessment of 

epistemological relevance i$ a .critical first step in evaluation* Though 

* { * 
the techniques of the evaluation researcher and the naturalistic investigator 

are qui t£ 'different, both aim at clarifying the nature of the evaluartd and 1 

surfacing the concerns of potential users of evaluation information, 

Epistemolqgical Rigor * 

- When we discus^ the properties of a 1 researchable 1 problem, we are 
speaking- of epistemological rigor* As was the case with epistemological 

<elevance t this notion of rigor is important regardless of the particular 
\ * * i 

philosophical orientation of the researcher or evaiuator* However, 'scientific 1 
and ^naturalistic 1 '\£ncfuirers assess the dimension of epistemological rigor in 
quite, different ways- ' 1 

Wheije evaluation is viewed- as an extension of scientific research, the 

. : 

, ^assessment 'of episte&ological rigor is quite straightforward* Here, rigor 
^effers ,to the extent -tb which evaluation questions are cast in a form that is 
irfeasu^aple or ^testabie* Assessment of 'rigor is largely context-independent 

'.and a firioai .y X %' involves Adequately specifying the ejnpirical referents for 
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and' the connections between variables contained in the "statement of the research 

or evaluation problem. Darcy (1979 , 19&0) illustrates this process in the 

'context of vocational" education outcomes evaluation. Beginning with a 

meaningful 6utcqme statement, he explains that (1) this' statement must be " 

translated into a. hypothesis with careful specification of dependent anE ' 

independent variables, (2) the entity on which the outcome, is to be observed 

must be delineated, and (3) empirical indicators of the outcome must b$ 

specified. Outcome statements so formulated meet the test of epistemological 

rigor. ' , ^ 

wHere evaluation is placed more in the tradition of ethnography (e.g*, 

Guba, 1978; P^tton, 1980; Filstead, 1979) tijte determination of epistemalogical 

rigor is no less important/ yet the process is less apparent because problems 

are not so carefully circumscribed prior to the investigation. Here, the 

assessment of episte mo logical rigor is ^largely context-dependent and ja * 

post eriori . That is, epistemologifcal. rigor is not assessed at the outset of 

qjx investigation by determining whether or not a problem is ^jesearchable / rather 

rigor is assessed near the conclusion of the investigation by determining 

whether or riot a problem has been h adequately researched (investigated)* 

Naturalistic or ethnographic evaluation approaches begin with problems . 

that are not largely delimited. Hence^ the naturalistic evaluator defines 

epistemological rigor in terms of whether the limits to an % investigation Qf 

a problem have been reached^ Guba (1978) describes this process as one. pf ^ 

reaching "closure" by applying the criteria of "exhaustXQn of resources*," 

"saturation/! 1 "emergence of regularities/" and "overextension" t:o the activity* 
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'•of collecting information* If these signals. for closure appear, then the 
Naturalistic evaiuator is reasonably certain that epistemological rigor has 
been attained* , J * 

Both approaches focus on setting the boundaries that define a researchable 
problem. In the case of evaluation research, these boundaries are defined in 

:/ " 

terms' of conditions for stating the problen^ In naturalistic evaluation/ these 
boundaries are defined in terms of outer linvits that signal completion of an 
investigation* * «. 

Methodological Relevance 

If we accept the proposition that .different -kinds of evaluation questions 
require different kinds of methodologies, then the question of methodological 
relevance can be asked as* M Is this panticular'method (or model) appropriate 
to the questions that I am trying to answer?" Assessir/g methodological 
relevance is largely a matter of determining the tradeoffs, in terms of 
strengths and weaknesses, of methods that are available to the evaluator,' # 

Questions of methodological relevance are largely means-end questions. 
It is only after knowing what we are trying to discover that we can decide 
how to proceed* For example/ if we wish to test causal hypotheses, then we- 
might choose an experimental design* If we wish to act as the surogate eyes 
and ears of decision makers who desire information about what really takes 
place in a program, then we might choose a case study approach* ^ 

Determining me^iodologic^l relevance requires (1) a review of the 
conditions which must be present to facilitate the use of a given method 
and (2) a careful consideration of the intrinsic strengths and weaknesses 
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of the method, \For Sample , in order to make use of an experimental or 
qua si- experimental evaluation design, conditions such, as the following must 
obtain: a clear, precise statement of intended program results; a reasonably 
'controlled 1 program setting; a reasonably 'uniform 1 treatment across 
participants and over time; a large enough sample; and an ability to select 
ajid assign individuals randomly to treatment and control groups, or the 
availability of a comparison group* Likewise, the measurement of program 
effects by means of objective, standardize^ instruments such as aptitude tests, 
achievement' tests, or attitude scales requires that there be a e program logic 
exhibiting valid linkages between the program's goals, the treatment delivered, 
and the instruments used to measure <3utcomes. 

To understand the second objective — the process of weighing the intrinsic 
merits of a given technique — consider the following review of the technique of 
documentary analysis. This method involves the analysis of written program 
materials — e.g., interim repo'rts^ "internal memoranda, activity logs', etc.— 
\o gain a clearer' insight of program planning and operation. Its strfi^ths 

are that it is entirely unobtrusive and nonreactivei that th"' documents 

/ . - 

themselves are unchd^gihg and express the perspectives 6f their authors in the 

/ . 
authors' own nat\^al language. On the othtfr hand, smcmg its weaknesses are 

that the document may not be representative, it is usually uniperspectival , 

represents yhique events, ^hd may be temporally and spatially specific. In 

a similar way, every method available to the evaluator-can be scrutinized for , 

its intrinsic adequacy or merit. \ 

* The activity of determining methcKjplogical relevance is clearly not a 

simple process* The choice of one method over another involves tfce evaluator 
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I tfhd other parties to the evaluation in a series of trade-off s' for which no 
set of rules will suffice. It; may be tempting to say .that the determination 
o£ relevance should be based on the principle of maximizing utility* But 
that raises the question of how we are to measure utility and for whom, A 
more plausible approach, may be to argue that instead of seeking optimal or 
maximal solutions to, the problem of methodological relevance* we .should adopt 
a strategy of "satisf icing" (Simon, 1976) — choosing methods which are not 
necessarily the best but 'good 'enough 1 given the goals of the evaluation/ 
the limitations of the methods themselves, the problems inherent in the 
particular evaluation situation/ and the needs of relevant parties to the 
evaluation* 

Questions of methodological relevance naturally raige the possibility 

of combining methods in a single study* The rationale for the use of multiple 

methods is captured nicely by Webb, ejt al^ (1966) : ' 

■ " Once a proposition has been confirmed by 6wo or more 

1 J measurement processes the uncertainty of its interpretation 

is greatly xetfuced* The most persuasive evidence comes 
* * 

through a triangul ation of measurement processes* (p. 3/ , 
emphasis added) 

Dejizin (1978) further suggests that, there are four, types r ,of triangulation 
available: (1) data triangulation— using a variety of data sources/, (2) 
investigator triangulation — using several different evaluators, (3) methodological 
triangulation— using several different methods to examine the same questions so 
that the flaws oi one mettiod can be compensated for by, the strengths of other *■ 
methods, and (4) theory triangulation — using multiple perspectives to interpret ■ 

■ .. V ■ 
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the same set of objectives. However, though the rationale may be veil- 

\ * 

established^ actually effecting^ methodological mix in a given study is a 

complicated matter' (cf* Trend, 19y9) requiring a great deal more investigation* 



Methodological Rigor 

* Unlike methodological relevance Which is aij assessment of instrumental 

worth, methodological rigor is a determination of intrinsic merit* When 

i *' 

weyask, "Is this method/evaluatioh-des^gn/type o£ datfa analysis rigorous?"' 

we are asking whether it meets certain agreed upon standards for a 'good'_ or 

■ ■ • 

'trustworthy 1 method, a 'sound 1 design*,^ or an 'appropriate! type of data 
analysis ]l 

Until very recently, ttfe only available and agreed tjpon standards for 
methodological rigor were the canons* for what constituted rigorous Scientific 
inquiry. For example, in their review^f federal evaluation studied 
Bernstein and Freeman (1975) developed a composite index of scientific • 
standard^ for measuring the quality (rigor) of evaluation., research* Their 



/ rating Scheme is shown be'lov in iable 1. 
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The Bernstein and Freeman index is fairly representative of the types of 
^standards currently in use for judging the methodological rigor "of both - 
- researched evaluation studies • Similar sets of standards are commonly 
employed in assessing the 'internal arid external validity of *re Search designs 
(Campbell 'and Stanley. 1966; Cook and Campbell, 1979), the psychometric' 
properties of ■ measurement devices (Guilford, .1954? Nunnaily-, 1978), etc* 

* < 
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Standards for* assessing the , methodological rigor of evaluation studies • 
conducted in a naturalistic or ethnographic mode are far Jfeess well developed- 

■ * \ 

A recent paper by Guba (In press ) represents one o^"€he first attempts to l 



specify standards for naturalistic studies. Table 2 below displays the 

criteria which Guba proposes for assessing the methodological rigor 

("trustworthiness"') of* naturalistic inquiries* Guba defines the naturalistic 

investigator's analog for criteria' such as objectivity ( f, conf irmability") , 

reliability 0' dependability") , generalizability ( M transferability 1 ' ) , anc^ 

f internal validity ("credibility") and lists and 'briefly explains methods - h% 

- that might be usegi to determine whether these criteria have been met, 
r • 

, 1 : 

Insert Table 2 about hare * ' * * k , 



Efforts such asrthis to specify the standards for judging not only'the design 

but the product of naturalistic inquiries are indispensable in view*df the 
t 

growing interest in, the use of naturalistic and ethnographic;- methods. 

Relationships Between Rigor and Relevance " 

The 'preceding four categories of rigor arid relevance — epistemological 

rigor, epistemoldgical relevance , methodological releyancfe* and. methodological 

rigor — have been presented in their most logical sequence, It should be 
** ' * * 

* * ' •j - 

apparent that efforts to frame a question in a rigorous waysjafould commence 
only ^after it is determined what it is that we are askin^Jptikewise (assuming 



the existence of standards for judging the rigor of both quantitative and 

* w - > 

qualitative methods)/ it is reasonable to believe that * questions of which 
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method to use can be, settled before examining how these methods might be 

used (or whether they have been used) in the most rigorous fashion* ^ ' # 

This analysis of rigor and relevance has also demonstrated that these 

notions are not necessarily inversely related* £n other wordsi an increase 
> 

Sri r4levance 'need not result in a decrease in rigor and "Vice-versa* To be 
sure/ the four'dimensions of . rigor and Relevance are not orthogonal. For, 

' •■" ' ' \ ■ ' 

example/ an assessment Of episfcemological relevance *informs the assessment 

of jnetho'dologicai ^relevance* Nevertheless/ one need ^not always^tra^eof f 

* - - * 
rigor for relevance* m K " 

Tradeoffs between rigor and relevance ►frequently (and quite inappropriately) 

characterize the choice of evaluation and research methods* It-is argued that 

.qne jnust chopsr. between 'rigorous and relevant methods* This demonstrates a , 

confounding of the notions of methodological rigor' and methodological relevance. 

As was discussed above/ the relevance o£ any method can Be assessed with 

respect to the goals of the researcher evaluation* Methods are instrumental! tie 

■ f * t 

the suitability o'f a method for meeting _ thfi goals of inquiry determine its -„ 

relevance* Rigor is another matter* Once a Relevant method ha^been chosen/ 

steps* can be taken to ensure the rigorous' use of th^t method. The only^w^ll- 

tteveloped and agreed upon standards, for rigor kpply to the use of ^quantitative 

♦methods* We have only recently begun to investigate standards for rigor that 

govern the use of qualitative methods* It is not the case that Qualitative - 

methods ajre inherently non-rigorous 4and herfbe/ somehow morfe relevant)'/ but * 

* ^ ^ 

that/ at present/ we are uncertain of bow to judge whether they have been" 

f >■ " 

used in* a^ rigorous fashion. ; 1 
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Rigor and Relevance in Vocational Education Evaluation 

s 

4 

Given the diversity of approaches to and methods for evaluating 
vocational education programs, it 'is hazardous to offer general statements 
regarding the extent to which the enterprise of vocational education * 
evaluation is. 1 addressing these questions of rigor and -relevance>. However, it 
is commonplace to find questions of rigor an§ relevatfce addressed to varying 

degrees within the context of particular methods or approaches. From these 

^ 

discussions there emerge several central tendencies which are discussed below. 
Epistemologic&l relevance is emerging as a primary concern after several 

'[ - 

years of evaluation efforts. For example, following a two-year study of 
vocational education outcomes by the National Center for Research in 



tional Education (Darcy, 1979, 1930) it was recommended that: 

Iii planning evaluation studies-/ care should be taken to ,o 
determine clearly what is to be -evaluated and what , * 



criteria, data, and evaluation standards ic^ .be used". 

(Darc^, i960, p* 70) 

This recommendation stems frdtn several findings of this study which point to 

shortcomings in assessing episteroolog^cal relevance*: (1) Terms "such a§ 

'outcomes,' 'outcome measures** 'pVogram goals,' and 'program benefits' lack 
+ 

precise 'definition,, (2} It is not *clear what is being evaluated — outcomes, 

groups of students, programs, etc./ (3) There is^little appreciation of the 

* 

range* diversity* and complexity of possible outcomes* and (4) The relative 

i f * 4 f 

importance of outcomes vis-a-vis otfier types of evaluation has not been well 

^addressed. 



Rigor and Relevance 



13 



The issue of epistemological relevancy has-been alluded to in other 
* * *t* * 

ways as well. In discussing the collection of evaluative^ data by^means o'f . > 
a standardized vocational, education data system^jjriewes (1978) no j te<£^thaJt 
answering Questions of why' data- -are to be collected (a dimension of assessing'* 1 
epistenological relevance) will determine the use and'utility of such a L 
system, Kievit (1978) sought to lay out a rationale for linking kin4s of y**- 
evaluative data to the values perspectives of potential usters, thereby* 
addressing the question of why^vocational education programs are in need 
of evaluation, > . . 

Determining episte mo logical rigor has always been, and will likely 
remain, a major concern of vocational evaluators. For example, techt's^ 
(1974) discussion of indicators of 'vocational program success gan be* viewed 
largely as an attempt-to address questions o£. epi^temological rigor in^ the 
definition and ^measurement of those indicators. Mdst recehtly, the link 
between epivtemo^gical rigor and relevance has been demonstrated in the 
vocational education outcomes study noted earlier. The study attempted to 



document epistemQlogical relevance for outcomes evaluation 



fcy^r 



equiring 



(if a clear rationale for the choice of art outcome, (2) eyi^oge of the 
appropriateness of^a*g^ven outcome as a basis for program evaluation, 
(3) illustration of the potential impact of results, .and (4) identification 
of relevant audiences for evaluative information. As noted earlier, the 
study thefl addressed epistemological rigor by indicating how outcome 
statements are to be translated into empirical measures of outcomes* Owing 
to the relatively recent iniportation of qualitative techniques to vocational 



-i r 
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evaluation; thqre are no commentaries on procedures for establishing 

epistemolo^ical rigor in ethnographic or naturalistic .vocational evaluation 

studies. * * 
i 

Hbwever, as alternative methodplogies for evaluating social programs 

have' found their way into evaluations of vocational programs, discussions 

of methodological relevance have emerged. Bolland (1979) , for example, 

briefly addresses the question of methodological relevance by listing the 

relative strengths and weaknesses of various data gathering techniques in 

her review of vocational education outcomes studies, Grasso (1979) 

discusses the suitabilit^ of impact Evaluation for meeting the evaluation 

requirements spelled out in the 1-976 vocational education legislation, , 

Spirer -(1980) points -'toWie utility of the case stiidy method in vocational 

edusation evaluation, "Bonnet (1979) discusses alternative, "methods for 
, ^ 

measuring the outcomesjot career education in view of the outcome goals 

set by the Qffice 6f Ca^r Education* Finally/ Rif^el (1980) recently 

offered, a very reflexive presentation ©h. the utility of the case study 

approach in the Vocational Education Study, and Pearsol (1980)' commented on 

* 

'the' jjiylicati^is^of ooipbining quantitative and qualitative methods- 
Methodological rigqr h&s perhaps been the mast frequently addressed 
aspect of rigor and relevance in vocational educatioh evaluation. Most, if 



/ai^ce 



not £11, of these discussions are concerned wi£h specifying standards for 
scientific rigor as it is commonly-perceived in the research community. 



r? 



Hence, Holland (1979) specifies eight basic components of a sound Research 



report, Morell (1979J and Franchak. an<f Spirer (197&) address design and 
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statistical' issues iii the application of follow-up res.earch to vocational 

t • 

evaluation. Borgatta % {1979) discusses the requirements for. 'good 1 . 

experimental and quasi-expefiroental designs* V 

Several papers address questions of methodological rigor and relevance 

simultaneously in reviewing , a particular research technique*" Pucel (1979) 
■ ' t / 

addresses questions of me thcfdo logical relevance by pointing to the types of t 
questions "that can be. answered trough longitudinal studies* He also focuses 
■ on aspect^ pf methodological rigor in the use of the method (e*g w solving * 
problems in implementation! specifying types of data to be collectedi eUtc, ) , 

Likewise, Franchak, et al, (1980) seek to demonstrate methodological relevance^ 

^ „ , r " ^ <£* 

by^ linking* the use of longitudinal methods to critical data needs in vocational 
educatipn, and they address problems of rigor in reviewing basic strategies 

* ; > • - 

and procedures £ar Ibngitudinal studies. Similarly, several publications in 
the Career Education Measurement Series (e,g*, McCaslin* et al * , 1979? 
McCaslin and- walker, 1979) address both methodological rigor and relevance in 
discussing the selection, evaluation, and design, of instruments to evaluate 
career education*, * " * 

In summary, the importance of addressing the issues of epi^temological 
**and methodological rigor and relevance can be seen in Lee's (1979) discussion 
of the factors governing the use of evaluation data. Lei identifies the 
following five factors: (1) availability (making- evaluation data available 
4 to users itt a, way that can .be readil^ understood) , (2) reliability, (3) 
credibility, (4) utility {collecting, analyzing, and interpreting evaluation 
data in view of their potentifti uses), and (5) consistency (collecting, 
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analyzing, and making available data within the boitndar ies of possible 
action). It is possible to recast these factors as functions of addressing 
rigor and relevance in evaluation studies* Thus* failure to address ' + t ♦ 

qi&stions of apis^emological relevance may lead to problems in consistency 
and utility; failure to address questions of e.pistemologi + cal ri^or and questions 
of methodological rigor may result in problems with reliability and credibility; 

finally, failure to address questions of methodological relevance may lead po ' 

' P ' 

problems with utility and. availability. 

Avenues for Future Study * 

Xll four dimensions of rigor and relevance discussed in this ^aper 

warrant further attention by + the community of vocational education evaluators 

and researchers. Epistemological relevance — determining what i-s to be 

evaluated and why--must clearly be our* foremost concern. Premature' focus on 

the selection, of appropriate methods will likely encourage the approach of 

* + 

■solutions m search of problems. 1 That is, we may' attempt to fit existing 
{and new and developing) evaluation strategies to particular vocational 

education evaluation problems without first under standing what it is we wish 

+ * ■ 

to know and why* There should be n^ equivocating^ arguing that we have the 

■right 1 solution but the "wrong 1 problem is simply an argument for the* wrong 

solution. We should not hesitati^ to retreat* from solutions to make a more 

careful diagnosis of the problem. + ■ 

Attending ta epistemological rigor presents us with two different types 

of problems. It appears that we are fully in -possession o^ the knowledge of 
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what constitutes/ a testable or measurable problem from a positivistic 

■ J ' 

perspective, Wnat t we heed are mor^ attempts* such as that demonstrated in 
%he vocational /educatio'n outcomes {study, to apjbly this knowledge to particular 
evaluation questions. On the other hand/ as naturalistic inquiry becomes 
increasingly -/relevant to evaluations of vocational, education programs, we 
will need to/ devote our " efforts* tp specifying procedures for determining the 
boundaries of such investigations!. The lack of a priori constraints, 
characteristic of 'this Approach, (does not imply a total lack or regard for 

J ' 1 1 

constraints which demonstrate ri^or in the investigation of problems: , 

Methodological relevance — including both an assessment of the intrinsic 
merits yc methods and an investigation of the possibilities for combining 
methods/- -demand s our most careful attention, lefet the choice of methods 
become^ simply a matter of what is 'currently in vogue. We must guard against 
'the normative appeal of certain established^methods as being the most., (or 
the only) 'rational 1 strategies and investigate the contextual limits 

■7 v ' . • • 

governing the .scope of these strategies. We must be careful not to nastake 

/ . ' " ^ 

evidence of the inapplicability of certain methods as- simply problems with 

tmpl emen tat ion , ^ ^ 

Finally, ( in the area of methodological rigor, we lack.lit'tle knowledge 

of traditional .standards for assessing the scientific adequacy of quantitative 

methods and experimental designs. Yet, we are largely ignorant of how to ^ 

judge the merit of case studies* emergent designs) and similar methods and * 

■ 

tools associated with naturalistic or ethnographic inquiries.. 

In general, we need to become Tnore open and public about our discussions 
or rigor and relevance. There are relatively few^acoounts of the conduct of 
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.vocational educ££iorL inquiries that are reflexive, Rsflexivity refers to the 



vocational educ< 
papaicity of thoi 



thought to bend back upon itself, to become an object to itself 
(Ruby, 1980), To-be reflexive is hot the same as being self-conscious or 
jreflective. ttost evaluators and researchers are, probably self-conscious, yet 
that ;kind of awarenes^ remains private knowledge for the inquirer, detached 
from the product of his or her inquiry, there are relatively fev accounts of 
pj)quiry in which inquirers reveal the epistemo logical 'and axiological 
assumptions which caused them to choose a particular set of questions to 
investigate, to seek answers to those questions in a particular way, and, , 
finally, to present their findings in a part^ular way. By" engaging in this 
kind of, reflexivity about our research Snd evaluation, we sure more lively 
to address critical issues in rigor and relevance. 



TABLE 1 ; - ; 

Criteria for Assessing the Quality 
of ^valuation ^Res^a^cJ^* 



Criterion 



Sampling 



1 
0 



Rating 

systematic 

nonrandom, cluster, or nonsy sterna tic 



D^ta Analysis 



2 - quantitative 

1 - qualitative and quantitative 1 
0 - qualitative * ' 



Statistical 
Procedures 



Design 



SarcpUing ' 



Measurement 
Proce^jir^s 1 



4 - multivariate 

3 - descriptive 

2 ( ~ ratings from qualitative data 

1 - narrative data only 1 

0 - no systematic material 

3 - experimental or quasi-experimental 

with randomization and control groups 

2 - experimental or quasi-experimental without 

both randomization and control groups 
f - longitudinal or cross^sectional without 
control or comparison 

0 - descriptive , narirativ^ ^ 

'2 ~ representative 

1 - possibly representative \ 

0 - haphazard 

1 - judged adequate in face validity 

0 «- judged less than adequate in face.validity 



1 1, 



'*Bern:rtei&, .Freeman^ 1£74/ pp.. 100-101 



TABLE 2 

Criteria for Assessing the Trustworthiness 
of Naturalistic Inquiries* 



Aspect; of Method, 
Study, Procedure 

Truth Value 



Applicability 



Consistency 



Neutrality 



Naturalistic Term 



Credibility 



Transferability 



Dependability 



Con f irmabi 1 i ty 



.Methods for Determining 
Whether Criteria^e Met 

prolonged engagement at^ 
s'iter'Peer debriefing, ; 
Triangulation , .Member 
checks , Collection 
referential adequacy 
materials * 

Theorejtical/puxpQsive 
san^iiigV -Collection 
T, thickV descriptive data" . 

4 * 

-Overlap methods, Stepwise 
',r h ecplication, Establish 
''au^it*' trai^L . - 

Tri angulation^ 

Conf irfeabilirty audi^: 



*Guba, Tup reps , passim 
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